Collinear AI released a public set of 100 deterministic tasks for evaluating large language models on fixing real-world vulnerabilities. Each task involves cloning an open-source repository and patching specific CWEs without breaking existing functionality. The benchmark uses programmatic scoring and Dockerized environments, allowing anyone to reproduce results locally without API costs.
It is worth running if you are building defensive agents or comparing open-weight models against API providers. The catch is that the dataset is gated on Hugging Face, so you must accept access conditions before using the Harbor runner. It is not a substitute for full penetration testing, as it only covers a specific subset of known weakness patterns.
more: https://huggingface.co/blog/cwe-bench
my other side: ivo.blogspot.com
Leave a Reply